Is There a Problem With Evidence in Health Professions Education?

ABSTRACT: What constitutes evidence, what value evidence has, and how the needs of knowledge producers and those who consume the knowledge produced as evidence might be better aligned are questions that continue to challenge the health sciences. In health professions education (HPE), debates on these questions have ebbed and flowed with little sense of resolution or progress. In this article, the authors explore whether there is a problem with evidence in HPE using thought experiments anchored in Argyris' learning loops framework.From a single-loop perspective ("How are we doing?"), there may be many problems with evidence in HPE, but little is known about how research evidence is being used in practice and policy. A double-loop perspective ("Could we do better?") suggests expectations of knowledge producers and knowledge consumers might be too high, which suggests more systemwide approaches to evidence-informed practice in HPE are needed. A triple-loop perspective ("Are we asking the right questions?") highlights misalignments between the dynamics of research and decision-making, such that scholarly inquiry may be better approached as a way of advancing broader conversations, rather than contributing to specific decision-making processes.The authors ask knowledge producers and consumers to be more attentive to the translation from knowledge to evidence. They also argue for more systematic tracking and audit of how research knowledge is used as evidence. Given that research does not always have to serve practical purposes or address the problems of a particular program or institution, the relationship between knowledge and evidence should be understood in terms of changing conversations, as well as influencing decisions.

Describing the Landscape of Medical Education Preprints on MedRxiv: Current Trends and Future Recommendations.

PURPOSE: A preprint is a version of a research manuscript posted to a preprint server prior to peer review. Preprints enable authors to quickly and openly share research, afford opportunities for expedient feedback, and enable immediate listing of research on grant and promotion applications. In medical education, most journals welcome preprints, which suggests preprints play a role in the field's discourse. Yet, little is known about medical education preprints, including author characteristics, preprint use, and ultimate publication status. This study provides an overview of preprints in medical education to better understand their role in the field's discourse. METHOD: The authors queried medRxiv, a preprint repository, to identify preprints categorized as "medical education" and downloaded related metadata. CrossRef was queried to gather information on preprints later published in journals. Data were analyzed using descriptive statistics. RESULTS: Between 2019 and 2022, 204 preprints were classified in medRxiv as "medical education," with most deposited in 2021 (n = 76, 37.3%). On average, preprint full-texts were downloaded 1,875.2 times, and all were promoted on social media. Preprints were authored, on average, by 5.9 authors. Corresponding authors were based in 41 countries, with 45.6% in the United States, United Kingdom, and Canada. Almost half (n = 101, 49.5%) became published articles in predominantly peer-reviewed journals. Preprints appeared in 65 peer-reviewed journals, with BMC Medical Education (n = 9, 8.9%) most represented. CONCLUSIONS: Medical education research is being deposited as preprints, which are promoted, heavily accessed, and subsequently published in peer-reviewed journals, including medical education journals. Considering the benefits of preprints and the slowness of medical education publishing, it is likely that preprint depositing will increase and preprints will be integrated into the field's discourse. The authors propose next steps to facilitate responsible and effective creation and use of preprints.

Development and Initial Validation of the Shame Frequency Questionnaire in Medical Students.

PURPOSE: This study presents the steps taken to develop and collect initial validity evidence for the Shame Frequency Questionnaire in Medical Students. METHOD: The instrument was based on a 7-step survey design; validity evidence was collected from content, response process, internal structure, and relationship to other variables. A literature review and qualitative interviews led to the design of the initial 16-item scale. Expert review and cognitive interviewing led to minor modifications in the original structure. Initial pilot testing was conducted in August 2019 in Uniformed Services University (USU) medical students; reliability assessment and exploratory factor analysis were performed. The revised 12-item scale was tested in January 2022 in Duke University School of Medicine medical students; reliability assessment, exploratory factor analysis, and correlation analysis with depression, burnout, anxiety, emotional thriving, and emotional well-being were performed. RESULTS: A total of 336 of 678 USU students (50%) and 106 of 522 Duke students (20%) completed the survey. Initial exploratory factor analysis of the USU data revealed 1 factor (shame), and 4 items were dropped from the scale according to predefined rules. Subsequent exploratory factor analysis of the Duke data revealed 1 factor; no further items were removed according to predefined rules. Internal consistent reliability was 0.95, and all interitem correlations were less than 0.85 for USU and Duke samples. As predicted, mean shame scale scores were positively correlated with anxiety (r = 0.54, P < .001), burnout (r = 0.50, P < .001), and depression (r = 0.47, P < .001) and negatively correlated with emotional thriving (r = -0.46, P < .001) and emotional recovery (r = -0.46, P < .001). CONCLUSIONS: The Shame Frequency Questionnaire in Medical Students is a psychometrically sound instrument with strong internal reliability and multisource validity evidence, supporting its use in studying shame in medical students.

Progression of Self-Directed Learning in Health Professions Education: Clarifying Terms and Processes.

As the complexity ofmedical knowledge and clinical practice continues to grow, physicians and physicians-intrainingmust learn to identify gaps in their knowledge and understand and engage in self-directed learning (SDL) in pursuit of academic goals and improved clinical performance. There is a lack of consensus, however, on the precise definition of SDL and how it relates to self-regulated learning (SRL) and co-regulated learning (CRL). We propose a conceptual framework for understanding the progression of a self-directed learner in the health professions and describe the interplay of SRL and CRL.

Faculty and Student Perceptions of Unauthorized Collaborations in the Preclinical Curriculum: Student or System Failure?

PURPOSE: Unauthorized collaboration among medical students, including the unauthorized provision of assistance and sharing of curricular and assessment materials, is a reported problem. While many faculty view such sharing as academic dishonesty, students do not always perceive these behaviors as problematic. With the trend toward more small-group and team-based learning and the proliferation of resource-sharing and online study aids, collaboration and sharing may have become a student norm. This multi-institutional, qualitative study examined faculty and student perceptions of and student motivations for unauthorized collaboration. METHOD: Using a constructivist approach, the authors conducted scenario-prompted semistructured interviews with faculty and students in the preclinical curriculum. Participants were asked to reflect on scenarios of unauthorized collaboration and discuss their perceptions of student motivation and the influence of personal or environmental factors. The authors performed inductive thematic analysis of the interview transcripts using open and axial coding followed by abstraction and synthesis of themes. RESULTS: Twenty-one faculty and 16 students across 3 institutions were interviewed in 2021. There was variation in perceptions among faculty and among students, but little variation between faculty and students. Both participant groups identified the same 3 areas of tension/themes: faculty/curriculum goals vs student goals, inherent character traits vs modifiable behavioral states, and student relationships with their peer group vs their relationships with the medical education system. Student behaviors were perceived to be influenced by their environment and motivated by the desire to help peers. Participants suggested cultivating trust between students and the education system, environmental interventions, and educating students about acceptable and unacceptable behaviors to prevent unauthorized collaboration. CONCLUSIONS: Given the various tensions and positive motivations behind unauthorized collaborations, institutions should consider explicitly preparing students to make thoughtful decisions when faced with competing priorities in addition to developing mitigation strategies that address the environment and its interactions with students.

Using Self-Regulated Learning Microanalysis to Examine Regulatory Processes in Clerkship Students Engaged in Practice Questions.

Introduction: Self-regulated learning is a cyclical process of forethought, performance, and self-reflection that has been used as an assessment tool in medical education. No prior studies have evaluated SRL processes for answering multiple-choice questions (MCQs) and most evaluated one or two iterations of a non-MCQ task. SRL assessment during MCQs may elucidate reasons why learners are successful or not on these questions that are encountered repeatedly during medical education. Methods: Internal medicine clerkship students at three institutions participated in a SRL microanalytic protocol that targeted strategic planning, metacognitive monitoring, causal attributions, and adaptive inferences across seven MCQs. Responses were transcribed and coded according to previously published methods for microanalytic protocols. Results: Forty-four students participated. In the forethought phase, students commonly endorsed prioritizing relevant features as their diagnostic strategy (n = 20, 45%) but few mentioned higher-order diagnostic reasoning processes such as integrating clinical information (n = 5, 11%) or comparing/contrasting diagnoses (n = 0, 0%). However, in the performance phase, students' metacognitive processes included high frequencies of integration (n = 38, 86%) and comparing/contrasting (n = 24, 55%). In the self-reflection phase, 93% (n = 41) of students faulted their management reasoning and 84% (n = 37) made negative references to their abilities. Less than 10% (n = 4) of students indicated that they would adapt their diagnostic reasoning process for these questions. Discussion: This study describes in detail student self-regulatory processes during MCQs. We found that students engaged in higher-order diagnostic reasoning processes but were not explicit about it and seldom reflected critically on these processes after selecting an incorrect answer. Self-reflections focused almost exclusively on management reasoning and negative references to abilities which may decrease self-efficacy. Encouraging students to identify and evaluate diagnostic reasoning processes and make attributions to controllable factors may improve performance.

Educator's blueprint: A how-to guide on survey administration.

In this paper, we take the lessons learned from designing a survey and collecting validity evidence and prepare to administer the survey for research. We focus specifically on how researchers can reach individuals in the target population, methods of contact and engagement, evidence-informed factors that enhance participation, and recommendations for follow-up with nonrespondents. We also discuss the challenges of survey administration and provide guidance for navigating low response rates. Surveys are a common tool used to evaluate educational initiatives and collect data for all types of research. However, many clinician educators conducting survey-based evaluation and research may struggle to efficiently administer their survey. As a result, they often struggle to obtain appropriate response rates and thus may have difficulty publishing their survey results. Previous papers in this series focused on the initial steps of survey development and validation, but it is equally important to understand how best to administer your survey to obtain meaningful responses from a representative sample.

Expanding Interdisciplinarity: A Bibliometric Study of Medical Education Using the Medical Education Journal List-24 (MEJ-24).

Introduction: Interdisciplinary research, which integrates input (e.g., data, techniques, theories) from two or more disciplines, is critical for solving wicked problems. Medical education research is assumed to be interdisciplinary. However, researchers have questioned this assumption. The present study, a conceptual replication, clarifies the nature of medical education interdisciplinarity by analyzing the citations of medical education journal articles. Method: The authors retrieved the cited references of all articles in 22 medical education journals between 2001-2020 from Web of Science (WoS). We then identified the WoS classifications for the journals of each cited reference. Results: We analyzed 31,283 articles referencing 723,683 publications. We identified 493,973 (68.3%) of those cited references in 6,618 journals representing 242 categories, which represents 94% of all WoS categories. Close to half of all citations were categorized as "education, scientific disciplines" and "healthcare sciences and services". Over the study period, the number of references consistently increased as did the representation of categories to include a diversity of topics such as business, management, and linguistics. Discussion: Our study aligns with previous research, suggesting that medical education research could be described as inwardly focused. However, the observed growth of categories and their increasing diversity over time indicates that medical education displays increasing interdisciplinarity. Now visible, the field can raise awareness of and promote interdisciplinarity, if desired, by seeking and highlighting opportunities for future growth.

The Effect of Assessments on Student Motivation for Learning and Its Outcomes in Health Professions Education: A Review and Realist Synthesis.

PURPOSE: In health professions education (HPE), the effect of assessments on student motivation for learning and its consequences have been largely neglected. This is problematic because assessments can hamper motivation and psychological well-being. The research questions guiding this review were: How do assessments affect student motivation for learning in HPE? What outcomes does this lead to in which contexts? METHOD: In October 2020, the authors searched PubMed, Embase, APA PsycInfo, ERIC, CINAHL, and Web of Science Core Collection for "assessments" AND "motivation" AND "health professions education/students." Empirical papers or literature reviews investigating the effect of assessments on student motivation for learning in HPE using quantitative, qualitative, or mixed methods from January 1, 2010, to October 29, 2020, were included. The authors chose the realist synthesis method for data analysis to study the intended and unintended consequences of this complex topic. Assessments were identified as stimulating autonomous or controlled motivation using sensitizing concepts from self-determination theory and data on context-mechanism-outcome were extracted. RESULTS: Twenty-four of 15,291 articles were ultimately included. Assessments stimulating controlled motivation seemed to have negative outcomes. An example of an assessment that stimulates controlled motivation is one that focuses on factual knowledge (context), which encourages studying only for the assessment (mechanism) and results in surface learning (outcome). Assessments stimulating autonomous motivation seemed to have positive outcomes. An example of an assessment that stimulates autonomous motivation is one that is fun (context), which through active learning (mechanism) leads to higher effort and better connection with the material (outcome). CONCLUSIONS: These findings indicate that students strategically learned what was expected to appear in assessments at the expense of what was needed in practice. Therefore, health professions educators should rethink their assessment philosophy and practices and introduce assessments that are relevant to professional practice and stimulate genuine interest in the content.

Well-being at a Military Medical School and Implications for Military Retention.

INTRODUCTION: Physical and psychological well-being play a critical role in the academic and professional development of medical students and can alter the trajectory of a student's quality of personal and professional life. Military medical students, given their dual role as officer and student, experience unique stressors and issues that may play a role in their future intentions to continue military service, as well as practice medicine. As such, this study explores well-being across the 4 years of medical school at Uniformed Services University (USU) and how well-being relates to a student's likelihood to continue serving in the military and practicing medicine. METHODS: In September 2019, 678 USU medical students were invited to complete a survey consisting of three sections-the Medical Student Well-being Index (MSWBI), a single-item burnout measure, and six questions regarding their likelihood of staying in the military and medical practice. Survey responses were analyzed using descriptive statistics, analysis of variance (ANOVA), and contingency table analysis. Additionally, thematic analysis was conducted on open-ended responses included as part of the likelihood questions. RESULTS: Our MSWBI and burnout scores suggest that the overall state of well-being among medical students at USU is comparable to other studies of the medical student population. ANOVA revealed class differences among the four cohorts, highlighted by improved well-being scores as students transitioned from clerkships to their fourth-year curriculum. Fewer clinical students (MS3s and MS4s), compared to pre-clerkship students, indicated a desire to stay in the military. In contrast, a higher percentage of clinical students seemed to "reconsider" their medical career choice compared to their pre-clerkship student counterparts. "Medicine-oriented" likelihood questions were associated with four unique MSWBI items, whereas "military-oriented" likelihood questions were associated with one unique MSWBI item. CONCLUSION: The present study found that the overall state of well-being in USU medical students is satisfactory, but opportunities for improvement exist. Medical student well-being seemed to have a stronger association with medicine-oriented likelihood items than with military-oriented likelihood items. To obtain and refine best practices for strengthening engagement and commitment, future research should examine if and how military and medical contexts converge and diverge throughout training. This may enhance the medical school and training experience and, ultimately, reinforce, or strengthen, the desire and commitment to practice and serve in military medicine.

Breaking Down Grit: Persistency and Flexibility in the Career Plans of Military Medical Students.

INTRODUCTION: The field of medicine is experiencing a crisis as high levels of physician and trainee burnout threaten the pipeline of future physicians. Grit, or passion and perseverance for long-term goals, has been studied in high-performing and elite military units and found to be predictive of successful completion of training in adverse conditions. The Uniformed Services University of the Health Sciences (USU) graduates military medical leaders who make up a significant portion of the Military Health System physician workforce. Taken together, an improved understanding of the relationships between burnout, well-being, grit, and retention among USU graduates is critical to the success of the Military Health System. MATERIALS AND METHODS: The current study was approved by the Institutional Review Board at USU and explored these relations among 519 medical students across three graduating classes. These students participated in two surveys approximately one year apart from October 2018 until November 2019. Participants completed measures on grit, burnout, and likelihood of leaving the military. These data were then merged with demographic and academic data (e.g., Medical College Admission Test scores) from the USU Long Term Career Outcome Study. These variables were then analyzed simultaneously using structural equation modeling to examine the relationships among variables in a single model. RESULTS: Results reaffirmed the 2-factor model of grit as both passion and perseverance (or interest consistency). No significant relationships emerged between burnout and other study variables. Sustained and focused interest was predictive of less likelihood of staying in the military. CONCLUSION: This study offers important insights into the relationship among well-being factors, grit, and long-term career planning in the military. The limitations of using a single-item measure of burnout and measuring behavioral intentions in a short time frame during undergraduate medical education highlight the importance of future longitudinal studies that can examine actual behaviors across a career lifespan. However, this study offers some key insights into potential impacts on the retention of military physicians. The findings suggest that military physicians who are most likely to stay in the military tend to embrace a more fluid and flexible medical specialty path. This is critical in expectation setting for the military to train and retain military physicians across a wide range of critical wartime specialties.

Profiles of Military Medical Students' Well-being, Burnout, and Retention.

BACKGROUND: Well-being concerns among medical students are more prevalent than their age-matched peers in the United States. It remains unknown, however, if individual differences in well-being exist among U.S. medical students serving in the military. In this study, we sought to identify profiles (i.e., subgroups) of well-being in military medical students and examine the associations between these well-being profiles and burnout, depression, and intended retention in military and medical fields. METHODS: Using a cross-sectional research design, we surveyed military medical students and then conducted latent class analysis to explore profiles of well-being, and applied the three-step latent class analysis method to assess predictors and outcomes of well-being profiles. RESULTS: Heterogeneity in well-being was identified among the 336 military medical students surveyed, portraying medical students' falling into three distinct subgroups: High well-being (36%), low well-being (20%), and moderate well-being (44%). Different subgroups were associated with different risks of outcomes. Students in the subgroup of low well-being were at the highest risk of burnout, depression, and leaving medicine. In contrast, students in the moderate well-being group were at the highest risk of leaving military service. CONCLUSIONS: These subgroups may be clinically important as burnout, depression, and intention to leave medical field and/or military service occurred with varying likelihoods among medical students across the different well-being subgroups. Military medical institutions may consider improving recruitment tools to identify the best alignment between students' career goals and the military setting. Besides, it is crucial for the institution to address diversity, equity, and inclusion issues that may lead to alienation, anxiety, and a sense of wanting to leave the military community.